Probabilistic Semi-Supervised Multi-Modal Hashing

In this paper, we propose a non-parametric Bayesian framework for multi-modal hash learning that takes into account the distance supervision (similarity/dissimilarity constraints). Our model embeds data of arbitrary modalities into a single latent binary feature with the ability to learn the dimensionality of the binary feature using the data itself. Given supervisory information (labeled similar and dissimilar pairs), we propose a novel discriminative term and develop a new Variational Bayes (VB) algorithm which incorporates that term into the proposed Bayesian framework. Let T = [X ,Y ] be the observed bi-modal data matrix where X = [x1,x2, ...,xd ]M×d and Y = [y1,y2, ...,yd ]N×d denote the first modal and the second modal data matrix respectively, and Z = [z1,z2, ...,zd ]K×d denotes the latent binary code matrix. In our VB framework, we truncate the length of the binary codes (K) and we set it to a finite but large number. If K is large enough, the analyzed multi-modal data using this number of bits, will reveal less than K bits. In order to incorporate the information of the similarity/dissimilarity constraints into the VB algorithm, we first define a regularizer for the binary code zi as